Introduction¶
Prosper is a peer-to-peer lending marketplace. At the time of writing, Prosper operates in the US and has issued loans worth over USD 22 billion to c. 1.4 million borrowers. Borrowers apply online for a fixed-rate, fixed-term loan between USD 2,000.00 and USD 40,000.00. Individuals, like you and I, and institutions, Sequoia Capital for example, invest in said loans. Prosper handles all loan servicing on behalf of the borrowers and investors.
The data set at hand has 113,937 observations (loans, if you like) and 81 variables; detailed information on the variables can be found here. Observations relate to the years 2005 to 2014, inclusive.
In [19]:
#upgrade `pandas`
!pip install --upgrade pandas
In [1]:
#import all packages and set plots to be embedded inline
import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)
warnings.filterwarnings(action='once')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from random import randint
from os import path
from contextlib import suppress
%load_ext autoreload
%autoreload 2
%reload_ext autoreload
%matplotlib inline
Default settings for plots¶
automate, as much as possible, the process of creating visualisations
why?
- it is efficient
- visualisations are consistent
how?
- create templates
In [2]:
#template no. 1
#default blue
default_blue = sns.color_palette('tab10')[0]
#default orange, if required
default_orange = sns.color_palette('tab10')[1]
#default palette
default_palette = sns.color_palette('tab10')
In [3]:
#template no. 2
'''
simple function to create `Figure` object
using matplotlib that contains an x-lab,
y-lab and title.
"Father Figure", if you like :)
3 params, all type `str`:
x_lab, y_lab and title
Please make sure all args passed to the
function are type `str`
return: None
'''
def create_fig(x_lab: str, y_lab: str, title: str):
"""create_fig function"""
try:
#fig size
plt.figure(figsize=(10, 6.18), dpi=216, frameon=False, clear=True)
#x-axis name
plt.xlabel(x_lab)
#y-axis name
plt.ylabel(y_lab)
#title
plt.title(title)
except ModuleNotFoundError:
print(f'Please `import matplotlib.pyplot as plt` and try again')
except:
print(f'Failed to create template')
raise
In [6]:
#template no. 3
'''
simple function to create `Figure` object
using matplotlib for sub-plots.
"Father Figure" for subplots :)
2 params, type int; the number of sub-plots.
@n_row -> #rows
@n_col -> #cols
return: fig and ax objects
'''
def create_sub(n_row: int=1, n_col: int=1):
"""create_sub function"""
try:
#fig, ax and figsize
fig, ax = plt.subplots(n_row, n_col, figsize=(10,6.18), dpi=216)
fig.tight_layout(pad=10.0)
return fig, ax
except ModuleNotFoundError:
print(f'Please `import matplotlib.pyplot as plt` and try again')
except:
print(f'Failed to create template')
raise
helper functions¶
In [4]:
#see if a df exists
'''
a simple function to see if a df exists
takes in 1 param: name of the df
Please do not pass the arg as a string
return: None
'''
def confirm_df_exists(df):
""" function confirm_exists """
if not df.empty:
print(f'This dataframe exists')
return
print(f'This dataframe does not exist')
In [6]:
#group data and find mean
'''
Function to calculate mean of
grouped data
Takes in 2 args: var_1 and var_2
Both args must be columns of a pandas DF
return: average of grouped data
'''
def group_avg(var_1, var_2):
""" function group_avg"""
if var_1 and var_2:
avg_gp_data = df.groupby(var_1)[var_2].mean()
return avg_gp_data
else:
print(f'Check that both arguments are columns of a pandas DF and try again')
In [7]:
#group data and find sum
'''
Function to calculate sum of
grouped data
Takes in 2 args: var_1 and var_2
Both args must be columns of a pandas DF
return: sum of grouped data
'''
def group_sum(var_1, var_2):
""" function group_sum"""
if var_1 and var_2:
sum_gp_data = df.groupby(var_1)[var_2].sum()
return sum_gp_data
else:
print(f'Check that both arguments are columns of s pandas DF and try again')
In [8]:
'''
function to create and sort a DF
3 params:
@x: pandas series #1
@y: pandas series #2
@ascending: bool, default T
return: pandas DF
'''
def custom_sort_df(x, y, ascending:bool=True):
"""function custom_sort_df"""
try:
dd = pd.DataFrame({k:v for (k, v) in zip(x, y)}, index=['Count']).transpose()
dd.sort_values('Count', inplace=True, ascending=ascending)
dd.reset_index(inplace=True)
return dd
except:
print(f'Check that x and y are pandas series and try again')
raise
In [2]:
'''
function to confirm that a file has been created
and/or exists in the current dir
one param: type `str`:
@file_name -> name of the file (include extension, please)
return: none
'''
def confirm_file_exists(file_name: str):
"""function confirm_file_exists"""
if path.exists(file_name):
print(f'File exists')
else:
print(f'Something went wrong. Investigate')
Load and inspect the data set¶
In [11]:
#load the data set
df = pd.read_csv('prosperLoanData.csv', sep=',')
confirm_df_exists(df)
In [15]:
df.shape
Out[15]:
In [16]:
df.duplicated().value_counts()
Out[16]:
In [17]:
df.info();
In [18]:
df.sample(randint(5, 15))
Out[18]:
In [ ]:
Structure¶
Overall¶
- 113,937 observations
- 81 variables
- 3 of type
bool - 50 of type
float - 11 of type
int - 17 of type
str
- 3 of type
- variable
LoanOriginationDateis of typestr - variables
OccupationandIncomeRangeare of typestr
Missing and null values¶
- some observations in
dfhave missing or null values- case(s) in point
- 100,596 under variable
GroupKey - 96,985 under variable
LoanFirstDefaultedCycleNumber
- 100,596 under variable
- will decide which ones to fill and/or drop on a case-by-case basis
- why?
- some variables are qualitative
- some variables apply to an observation conditionally
- example:
GroupKeyrepresents observations that fall under a criteria; the number of said groups cannot be reasonably expected to be 113,937
- example:
- why?
- case(s) in point
Duplicated observations¶
dfhas no duplicated observations
Multiple values for a variable¶
- observations in
dfhave 1 value per variable
Feature(s) of interest in the data set¶
- characteristics of Prosper's loans over time
- how many loans have been written
- what the average rate offered to borrowers is
- what the average amount of a loan is
- what the average customer looks like
- another way of saying it is: to whom does Prosper lend?
- what affects the rate offered to a customer (borrower, if you like)
- what affects the number of investors of a loan
Feature(s) in the data set that may help support investigation into feature(s) of interest¶
- variables
LoanOriginalAmount,LoanOriginationDateandBorrowerRatewill help answer the first question - variables
Borrowerrate,Term,BorrowerAPR,StatedMonthlyIncomeandEmploymentStatusDurationamong others will help answer the next two questions - variables
Borrowerrate,PercentFunded, andInvestorswill help answer the last question
Clean the data set¶
- will perform a chap-chap cleaning process viz:
- drop variables whose missing values are above a threshold (in this case, 15%)
- drop observations whose missing values are above a threshold (in this case, 10%)
- change the type of variable
LoanOriginationDatetodatetime - change the type of variables
OccupationandIncomeRangetocategory
In [12]:
#create a copy of the df
df = df.copy()
In [20]:
#drop variables whose missing values are above the threshold
df.drop(columns=df.columns[df.eq(0).mean()>0.85], inplace=True)
df.shape
Out[20]:
In [ ]:
#drop observations whose missing values are above the threshold
df.drop(index=df.index[df.eq(0).mean()>0.9], inplace=True)
In [ ]:
#save cleaned data set as a `.csv` file then load it under the variable `df`
#save data set as a `.csv`
df.to_csv('prosper_loans_data.csv', index=False, encoding='utf-8', sep=',')
#confirm that the`.csv` file exists
confirm_file_exists('prosper_loans_data.csv')
In [5]:
#load the clean df
df = pd.read_csv('prosper_loans_data.csv', sep=',')
confirm_df_exists(df)
In [11]:
#change the type of variable `LoanOriginationDate` to `datetime`
df.LoanOriginationDate = pd.to_datetime(df.LoanOriginationDate)
df.LoanOriginationDate.sample(randint(3, 5))
Out[11]:
In [12]:
#change the type of variables `Occupation` and `IncomeRange` to `category`
df.Occupation = df.Occupation.astype("category")
df.IncomeRange = df.IncomeRange.astype("category")
In [13]:
#handling categorical variables for ease of plotting
df_0 = {'LoanStatus' : ['Current', 'Completed', 'FinalPaymentInProgress', 'Past Due (1-15 days)', 'Past Due (16-30 days)',
'Past Due (31-60 days)', 'Past Due (61-90 days)', 'Past Due (91-120 days)', 'Past Due (>120 days)',
'Defaulted', 'ChargedOff', 'Cancelled'],
'IncomeRange' : ['$0', '$1-24,999', '$25,000-49,999', '$50,000-74,999', '$75,000-99,999', '$100,000+', 'Not displayed',
'Not employed'],
'ProsperRating (Alpha)' : ['AA', 'A', 'B', 'C', 'D', 'E', 'HR'],
'IsBorrowerHomeowner' : [True, False],
'IncomeVerifiable' : [False, True]
}
for i in df_0:
ordered_df = pd.api.types.CategoricalDtype(ordered= True, categories= df_0[i])
df[i] = df[i].astype(ordered_df)
In [14]:
df.info()
Generate a pre-EDA report using sweetviz¶
In [20]:
# pre_eda_rpt = sv.analyze(df)
# pre_eda_rpt.show_html('./Summary_EDA_Report.html')
##does not work; notebook does not support sweetviz :(
Univariate Exploration¶
Method ¶
- ask a question
- state what will be done to answer the question
- code
- do what was stated
- visualise
- state the findings
In [28]:
pd.DataFrame(df.LoanOriginalAmount.describe())
Out[28]:
Findings¶
- average amount borrowed is c. USD 8,337.00
- loans of amount USD 5k, 10k, 15k, 20k and 25k have higher-than-average frequency
- the range is USD 1,000.00 to 35,000.00
- Prosper's website says USD 2,000.00 to 40,000.00
- this could be, among others:
Prospercould have revised its loan policyProspercould have increased its product offering
- the CV is c. 0.75 (6245.80058/8337.01385); this suggests that majority of the observations are close to the mean
- majority of the amounts are in the range USD 1,000.00 to 15,000.00
- the mean is greater than the median (50%); this suggests a skew to the right
a histogram of the above...
In [29]:
#create a hist
create_fig('Amount borrowed (USD)','Frequency (Count)','Frequency distribution of amount(s) borrowed')
plt.hist(df.LoanOriginalAmount, bins=50, color=default_blue);
In [30]:
pd.DataFrame(df.BorrowerRate.describe())
Out[30]:
Findings¶
- average rate is c. 0.19%
- the range is zero to 0.5%
- does
Prosperoffer interest-free loans? - if yes:
- who do they lend to at that rate?
- why?
- what are the pre-requisites and terms?
- does
- the CV is c. 0.39 (0.074818/0.192764); this suggests that observations are more or less evenly distributed around the mean
- there are, however, rates that have higher-than-average frequency at the range 0.08% to 0.35%
- the mean is slightly greater than the median (50%); this suggests a slight skew to the right
a histogram of the above supports the findings...
In [31]:
#create hist
create_fig('Rate(s) of borrowing (%)', 'Frequency (Count)', 'Frequency distribution of borrower rates')
plt.hist(df.BorrowerRate, bins=100, color=default_blue);
In [29]:
#sum of loans by year
sum_loan_time = group_sum('LoanOriginationDate','LoanNumber')
In [30]:
#number of loans issued
y = [sum_loan_time.values[i] for i in range(len(sum_loan_time))]
#time (years)
x = [sum_loan_time.index[i].year for i in range(len(sum_loan_time))]
#create plot
create_fig('Time (Years)', '#Loans issued (tens of millions)', 'Number of loans issued YoY')
plt.bar(x, y, tick_label=x, color=default_blue)
plt.yticks(np.arange(0, 65000000, step=10000000));
In [34]:
#view basic descriptive stats of `sum_loan_time`
pd.DataFrame(sum_loan_time).describe()
Out[34]:
In [35]:
#time series plot for `sum_loan_plot` by the day
create_fig('Time (Days)', '#Loans issued (tens of millions)', 'Number of loans issued over time')
sum_loan_time.plot(kind='line',color=default_blue)
plt.xlabel('Time (delta = 1 day)');
Findings¶
the number of loans disbursed changes with time
- the number fluctuates
- there are seemingly regular but wide fluctuations according to the time series
- fluctuations in the periods Q4 2013 to Q2 2014 inclusive are remarkably wide
- there are little to no fluctuations starting Q4 2008 to Q3 2009, inclusive (this is exactly 12 months)
- the highest number of loans issued, in a year, before 2011 were issued in 2008 according to the bar chart
- there are seemingly regular but wide fluctuations according to the time series
- the number fluctuates
there appears to be an exponential trend
- trend shows that number of loans issued appears to increase exponentially despite the fluctuations
findings from summary statistics
- the average number of loans issued, per annum, between 2005 and 2014 inclusive is c. 4.3 million
- the range is 1 to c. 63 million
- the standard deviation is large (c. 7.5 million); it follows that the variance (the fluctuations discussed above) is quite large
What, really, happens:
- before 2007
- during Q4 2008 to Q3 2009, inclusive
In [36]:
#visualise said timelines
create_sub(2,1)
plt.suptitle('Time series plots for outlier data')
#sub-plot #1: loans issued pre-2007
plt.subplot(2, 1, 1)
sum_loan_time.plot(kind='line',color=default_blue, xlim=['2005-12-31','2008-01-01'], title='#Loans issued (2005 - 2007)', fontsize=9)
plt.xlabel('Time (delta = 1 day)')
plt.ylabel('#Loans issued');
#sub-plot #2: loans issued during Q4 2008 to Q3 2009
plt.subplot(2,1,2)
sum_loan_time.plot(kind='line',color=default_blue, xlim=['2008-09-01','2009-09-30'], title='#Loans issued (Q4 2008 - Q3 2009)', fontsize=9)
plt.xlabel('Time (delta = 1 day)')
plt.xticks(['2008-09', '2008-12', '2009-03', '2009-06', '2009-09'])
plt.ylabel('#Loans issued');
What happens before 2007?¶
- nothing, for the most part
- activity begins at the beginning of Q2 2007
- little activity; less than a million loans, in total, are issued
What happens during Q4 2008 to Q3 2009, inclusive?¶
- nothing, for the most part
- activity ceases during the middle of Q4 2008
- said activity stays at zero until late Q2 2009
- little activity; less than fifteen million loans, in total, are issued
Discussion on the distribution(s) of the variable(s) of interest¶
- variable
LoanOriginalAmount:- multi-modal; there are peaks at multiples of USD 5,000.00
- skewed to the right
- majority of the observations are close to the mean
- variables
LoanOriginationDateandLoanNumber:- have been processed to show the number of loans issued over time using a time series plot
- skewed to the left; suggests an exponential-type distribution
- variable
BorrowerRate:- multi-modal; there are peaks at multiples of 0.5%
- skewed to the right
- majority of the observations are close to the mean
are there any unusual points?¶
- yes, for the time series plot
- little to no activity during:
- 2005 to 2007
- Q4 2008 to Q3 2009
- little to no activity during:
was there a need to perform any transformations?¶
- yes
- variables
LoanOriginationDateandLoanNumberwere processed to show the number of loans issued over time
- variables
of the features investigated, are there any unusual distributions?¶
- none, to the best of my knowledge
have any operations to tidy, adjust, or change the form of the data been performed?¶
- yes
- variable
LoanOriginationDatewas converted to typedatetime
- variable
if so, why?¶
- to create a time series plot
carrying on with univariate exploration...¶
Question #4¶
To whom does Prosper lend?¶
- this is a broad question; emphasis on occupation, income level, home ownership status, state of residence and employment status
- plot a chart of:
- top ten occupations
- bottom ten occupations
- income groups
- home ownership
- top ten states of residence
- bottom ten states of residence
- employment status
- plot a chart of:
In [18]:
#1. occupation: top ten
y = df.Occupation.value_counts().values[:10]
x = df.Occupation.value_counts().index[:10]
#sort top 10
dd = custom_sort_df(x, y)
#create viz
create_fig('Count', 'Occupation', 'Top ten occupations of Prosper\'s customers' )
plt.hlines(y=dd.index, xmin=0, xmax=dd.Count, color=default_blue, linewidth=20)
plt.yticks(np.arange(len(x)), dd.iloc[:,0]);
Categories Other and Professional
Otheris ambiguous; not quite certain what it representsProfessional, according to the US bureau of labour statistics, appears to be a collective term
What happens when Other and Professional are excluded?
In [91]:
#2. occupation: top ten sans `Others` and `Professional`
y = df.Occupation.value_counts().values[2:12]
x = df.Occupation.value_counts().index[2:12]
#sort top 10
dd = custom_sort_df(x, y)
#create viz
create_fig('Count', 'Occupation', 'Top ten occupations of Prosper\'s customers (sans \'Other\' and \'Professional\')' )
plt.hlines(y=dd.index, xmin=0, xmax=dd.Count, color=default_blue, linewidth=20)
plt.yticks(np.arange(len(x)), dd.iloc[:,0]);
In [90]:
#3. occupation: bottom 10
y = df.Occupation.value_counts().values[-10:]
x = df.Occupation.value_counts().index[-10:]
#sort bottom 10
dd = custom_sort_df(x, y)
#create viz
create_fig('Count', 'Occupation', 'Bottom ten occupations of Prosper\'s customers' )
plt.hlines(y=dd.index, xmin=0, xmax=dd.Count, color=default_blue, linewidth=20)
plt.yticks(np.arange(len(x)), dd.iloc[:,0]);
In [40]:
#show top ten sans 2 unclear categories
y = df.Occupation.value_counts().values[2:12]
x = df.Occupation.value_counts().index[2:12]
top_ten = pd.DataFrame({k:v for (k, v) in zip(x, y)}, index=['Count']).transpose()
top_ten
Out[40]:
In [41]:
#show bottom ten
y = df.Occupation.value_counts().values[-10:]
x = df.Occupation.value_counts().index[-10:]
bottom_ten = pd.DataFrame({k:v for (k, v) in zip(x, y)}, index=['Count']).transpose()
bottom_ten
Out[41]:
Findings¶
Occupation¶
- an overwhelming majority of the top ten occupations are white collar
- a majority of the bottom ten occupations, that are not students, are blue collar
- half of the occupations in the bottom ten are students
In [88]:
#4. income Levels
y = df.IncomeRange.value_counts().values[:]
x = df.IncomeRange.value_counts().index[:]
#sort income levels
dd = custom_sort_df(x, y)
#create viz
create_fig('Count', 'Income level (USD, yearly)', 'Prosper\'s customers by income level')
plt.hlines(y=dd.index, xmin=0, xmax=dd.Count, color=default_blue, linewidth=20)
plt.yticks(np.arange(len(x)), dd.iloc[:, 0]);
In [ ]:
In [43]:
#show income levels
pd.DataFrame({k:v for (k, v) in zip(x,y)}, index=['Count']).transpose()
Out[43]:
Findings¶
Income level¶
- an overwhelming majority of loans are issued to those whose income level is above USD 25,000.00 p.a.
- the top 2 groups account for 55.8% of loans issued
- the top 4 groups account for 85.4% of loans issued
- top 4 income groups by loans issued:
$25,000-49,999$50,000-74,999$100,000+$75,000-99,999
- the category
Not displayedis ambiguous, as such, cannot be interpreted - prosper lends to unemployed applicants
- unemployed refers to the category
Not employed. Not sure if an employee can earn USD 0.00 p.a.; this article and this one have diverging views
- unemployed refers to the category
In [75]:
#5. home ownership status
#home ownership: values to plot
y = df.IsBorrowerHomeowner.value_counts().values[:]
x = df.IsBorrowerHomeowner.value_counts().index[:]
#create viz
create_fig('Is borrower a home-owner?', 'Count', 'Home ownership status of Prosper\'s customers')
plt.bar(x, y, color=default_blue)
plt.xticks(np.arange(2), ['No', 'Yes']);
In [45]:
#show home ownership status
pd.DataFrame({k:v for (k, v) in zip(x, y)}, index=['Count']).transpose()
Out[45]:
In [21]:
#6. location by state
#state top-ten: values to plot
y = df.BorrowerState.value_counts().values[:10]
x = df.BorrowerState.value_counts().index[:10]
#sort states
dd = custom_sort_df(x, y, ascending=False)
#create viz
create_fig('Place of residence (State)', 'Count', 'Prosper\'s customers by location (top 10 states)')
plt.vlines(x=dd.index, ymin=0, ymax=dd.Count, color=default_blue, linewidth=20)
plt.xticks(np.arange(len(x)), dd.iloc[:, 0]);
In [22]:
#state bottom ten: values to plot
y = df.BorrowerState.value_counts().values[-10:]
x = df.BorrowerState.value_counts().index[-10:]
#sort states
dd = custom_sort_df(x, y, ascending=False)
#create viz
create_fig('Place of residence (State)', 'Count', 'Prosper\'s customers by location (bottom 10 states)')
plt.vlines(x=dd.index, ymin=0, ymax=dd.Count, color=default_blue, linewidth=20)
plt.xticks(np.arange(len(x)), dd.iloc[:, 0]);
In [48]:
#show top ten
y = df.BorrowerState.value_counts().values[:10]
x = df.BorrowerState.value_counts().index[:10]
pd.DataFrame({k:v for (k, v) in zip(x, y)}, index=['Count']).transpose()
Out[48]:
In [49]:
#show bottom ten
y = df.BorrowerState.value_counts().values[-10:]
x = df.BorrowerState.value_counts().index[-10:]
pd.DataFrame({k:v for (k, v) in zip(x, y)}, index=['Count']).transpose()
Out[49]:
Findings¶
State¶
- the state of California has the most borrowers
- the state of North Dakota has the least borrowers
- a majority of the states with the least borrowers are from the so-called "fly-over country"
- all of the states in the top ten list have a large population
In [27]:
#7 employment status
#employment status: values to plot
y = df.EmploymentStatus.value_counts().values[:]
x = df.EmploymentStatus.value_counts().index[:]
#sort income levels
dd = custom_sort_df(x, y)
#create viz
create_fig('Count', 'Employment status', 'Employment status of Prosper\'s customers')
plt.hlines(y=dd.index, xmin=0, xmax=dd.Count, color=default_blue, linewidth=20)
plt.yticks(np.arange(len(x)), dd.iloc[:, 0]);
Employed is ambiguous; does it mean:
Full-time,Part-timeorSelf-employed- any combination of the three
Is it to be treated as its own category?
- Cannot tell from the description of variables or upon examination of variable
EmploymentStatus
Bivariate Exploration¶
Will follow the method used in univariate exploration¶
Question #5¶
What affects the BorrowerRate?¶
BorrowerRatewas discussed in Question #2- emphasis on term, APR, income level and duration of employment status
- scatter plots of
BorrowerRateagainst:- term of loans
- annual per-centage rate
- stated monthly income
- employment status duration
In [24]:
#1. term
#see correlation between `Term` and `BorrowerRate`
fig, ax = create_sub()
sns.stripplot(df.Term, df.BorrowerRate, color=default_blue)
#title
plt.title('How the term of a loan correlates to the rate')
#labels
plt.xlabel('Term (months)')
plt.ylabel('Rate (% pa, nominal)');
In [33]:
#correlation matrix of `Term` and `BorrowerRate`
corr_df = pd.DataFrame([df.Term, df.BorrowerRate]).transpose()
corr_df.corr()
Out[33]:
In [26]:
#jittered stripplot for `BorrowerRate` against `Term`
fig, ax = create_sub()
sns.stripplot(df.Term, df.BorrowerRate, jitter=0.25, size=3, ax=ax, linewidth=.5)
#title
plt.title('How the term of a loan correlates to the rate (jittered)')
#labels
plt.xlabel('Term (months)')
plt.ylabel('Rate (% pa, nominal)');
what happens when data points are jittered...¶
- the edges of the clusters are clearer
- an overwhelming majority of loans of all terms are in the range 0.05% to 0.35%
- values for
Term= 12 months are not as dense (tightly packed, if you like) as, say,Term= 36 months- majority of the tenure of the loans is 3 years or more
- there are unusually high and low rates at
Term= 36 months - correlation between
TermandBorrowerRate- the correlation co-efficient is c. 0.02; it shows very weak, positive correlation
In [32]:
#2. APR
#see correlation between `BorrowerAPR` and `BorrowerRate`
fig, ax = create_sub()
sns.stripplot(df.BorrowerAPR, df.BorrowerRate, color=default_blue)
#title
plt.title('How the APR of a loan correlates to the rate')
#labels
plt.xlabel('Rate (% pa, effective)')
plt.ylabel('Rate (% pa, nominal)')
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
In [31]:
#correlation matrix of `BorrowerAPR` and `BorrowerRate`
corr_df = pd.DataFrame([df.BorrowerAPR, df.BorrowerRate]).transpose()
corr_df.corr()
Out[31]:
Findings¶
Annual Percentage Rate¶
- there is a positive correlation between
BorrowerAPRandBorrowerRate- this, according to this article is the norm
- the positive correlation is very strong (c. 0.99)
- there is interesting behaviour at the extremes
- low effective rates have quite low nominal rates and vice versa
what happens when data points are jittered?
In [34]:
#jittered stripplot for `BorrowerRate` against `BorrowerAPR`
fig, ax = create_sub()
sns.stripplot(df.BorrowerAPR, df.BorrowerRate, jitter=1.5, size=3, ax=ax, linewidth=.5)
#title
plt.title('How the APR of a loan correlates to the rate (jittered)')
#labels
plt.xlabel('Rate (% pa, effective)')
plt.ylabel('Rate (% pa, nominal)')
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
what happens when data points are jittered...¶
- outliers present themselves
- example: the value that is below 0.1% while the rest are at least 0.1%
In [39]:
#3. stated monthly income
#see correlation between `StatedMonthlyIncome` and `BorrowerRate`
fig, ax = create_sub()
sns.stripplot(df.StatedMonthlyIncome, df.BorrowerRate, color=default_blue)
#title
plt.title('How the stated income of a borrower correlates to the rate')
#labels
plt.xlabel('Stated income (USD, monthly)')
plt.ylabel('Rate (% pa, nominal)')
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
In [14]:
#correlation matrix of `StatedMonthlyIncome` and `BorrowerRate`
corr_df = pd.DataFrame([df.StatedMonthlyIncome, df.BorrowerRate]).transpose()
corr_df.corr()
Out[14]:
Findings¶
Stated monthly income¶
BorrowerRateis clustered at the range 0.05% to 0.35% for all levels ofStatedMonthlyIncome- correlation is not clear from the visualisation
- will jitter the data points for visual clarity
- there is very weak, negative correlation (c. -0.09) according to the correlation matrix
- there are outliers at all levels of
StatedMonthlyIncome- there are a few unusually high and low rates
what happens whan data points are jittered?
In [50]:
#jittered stripplot for `BorrowerRate` against `StatedMonthlyIncome`
fig, ax = create_sub()
sns.stripplot(df.StatedMonthlyIncome, df.BorrowerRate, jitter=0.5, size=3, ax=ax, linewidth=.5)
#title
plt.title('How the stated income of a borrower correlates to the rate (jittered)')
#labels
plt.xlabel('Stated income (USD, monthly)')
plt.ylabel('Rate (% pa, nominal)')
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
what happens when data points are jittered...¶
- edges become clearer
- the upper bound at the 0.35% rate can be seen
- at lower rate levels (0.0% - 0.05%), there are more observations of low income than high income
- trends begin to appear
- data points form lines at 0.35%, 0.325%, 0.29%, 0.15% and 0.1%
- the bottom right of the cluster, that is, the region of high income and low rates, is quite dense
- the number of data points here apprears to be more-than-average
In [35]:
#4. employment duration
#see correlation between `EmploymentStatusDuration` and `BorrowerRate`
fig, ax = create_sub()
sns.stripplot(df.EmploymentStatusDuration, df.BorrowerRate, color=default_blue)
#title
plt.title('How the employment duration of a borrower correlates to the rate')
#labels
plt.xlabel('Employment status duration (days)')
plt.ylabel('Rate (% pa, nominal)');
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
In [18]:
#correlation matrix of `EmploymentDuration` and `BorrowerRate`
corr_df = pd.DataFrame([df.EmploymentStatusDuration, df.BorrowerRate]).transpose()
corr_df.corr()
Out[18]:
Findings¶
Employment Status Duration¶
BorrowerRateis clustered at the range 0.05% to 0.35% for most levels ofEmploymentStatusDuration- the cluster is dense at low to mid duration and sparse thereafter
- at the highest income levels, individual points do not overlap as much as at other levels
- correlation is not clear from the visualisation
- will jitter the data points for visual clarity
- there is very weak, negative correlation (c. -0.02) according to the correlation matrix
- there are outliers
- there are fewer unusually low than high rates
what happens when data points are jittered?
In [11]:
#jittered stripplot for `BorrowerRate` against `EmploymentDuration`
fig, ax = create_sub()
sns.stripplot(df.EmploymentStatusDuration, df.BorrowerRate, jitter=0.25, size=3, ax=ax, linewidth=.5)
#title
plt.title('How the employment duration of a borrower correlates to the rate (jittered)')
#labels
plt.xlabel('Employment status duration (days)')
plt.ylabel('Rate (% pa, nominal)');
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
what happens when data points are jittered...¶
- edges become clearer
- the upper bound at the 0.35% rate can be seen
- the lower bound is not quite as prominent as the upper one
- at lower rate levels (0.0% - 0.05%), there are more data points at low duration than high duration
- trends begin to appear
- data points form lines at 0.35%, 0.325%, 0.275% and 0.25%
- the left of the cluster, that is, the region of low duration, is quite dense
- the number of data points here apprears to be more-than-average
In [14]:
#see correlation between `BorrowerRate` and `PercentFunded`
fig, ax = create_sub()
sns.stripplot(df.BorrowerRate, df.PercentFunded, color=default_blue)
#title
plt.title('How the nominal rate of a loan correlates to the proportion of loan funded')
#labels
plt.xlabel('Rate (% pa, nominal)')
plt.ylabel('Proportion of loan funded (per unit)')
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
In [22]:
#correlation matrix of `BorrowerRate` and `PercentFunded`
corr_df = pd.DataFrame([df.BorrowerRate, df.PercentFunded]).transpose()
corr_df.corr()
Out[22]:
Findings¶
PercentFundedis clustered at the range 0.7 (70% funding) to 1 (100% funding) for all levels ofBorrowerRate- there is a clear line at 100% funding
- there is a cluster of points at the high rate level beyond which loans are funded only 100%
- the data points are all over the place, so to speak
- correlation is not clear from the visualisation
- will jitter the data points for visual clarity
- there is very weak, negative correlation (c. -0.03) according to the correlation matrix
- correlation is not clear from the visualisation
what happens when data points are jittered?
In [48]:
#jittered stripplot for `BorrowerRate` against `PercentFunded`
fig, ax = create_sub()
sns.stripplot(df.BorrowerRate, df.PercentFunded, jitter=0.25, size=3, ax=ax, linewidth=.5)
#title
plt.title('How the nominal rate of a loan correlates to the proportion of loan funded (jittered)')
#labels
plt.xlabel('Rate (% pa, nominal)')
plt.ylabel('Proportion of loan funded (per unit)')
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
what happens when data points are jittered...¶
- edges become clearer
- the line at the 100% proportion can be seen
- the cluster at the high rate level is prominent
- trends begin to appear
- data points form lines at regular intervals of the rate
- the data points above 100% proportion are as a result of jittering; please ignore these
In [25]:
#see correlation between `BorrowerRate` and `Investors`
fig, ax = create_sub()
sns.stripplot(df.BorrowerRate, df.Investors, color=default_blue)
#title
plt.title('How the nominal rate of a loan correlates to the number of investors in said loan')
#labels
plt.xlabel('Rate (% pa, nominal)')
plt.ylabel('Number of Investors');
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
In [16]:
#correlation matrix of `BorrowerRate` and `Investors`
corr_df = pd.DataFrame([df.BorrowerRate, df.Investors]).transpose()
corr_df.corr()
Out[16]:
Findings¶
Investorsis clustered at the range 0 to c. 400 for all levels ofBorrowerRate- the cluster is densest at the range 0 to 200
- there apprears to be more data points at the region of low rates and low number of investors
- there appears to be a lot of data points at
Investors= 0
- correlation is not clear from the visualisation
- will jitter the data points for visual clarity
- there is weak, negative correlation (c. -0.27) according to the correlation matrix
what happens when data points are jittered?
In [47]:
#jittered stripplot for `BorrowerRate` against `Investors`
fig, ax = create_sub()
sns.stripplot(df.BorrowerRate, df.Investors, jitter=0.25, size=3, ax=ax, linewidth=.5)
#title
plt.title('How the nominal rate of a loan correlates to the number of investors in said loan (jittered)')
#labels
plt.xlabel('Rate (% pa, nominal)')
plt.ylabel('Number of Investors');
#supress xticks; only the trend is required
with suppress(TypeError):
plt.xticks(np.arange(0, 0, 1), None);
what happens when data points are jittered...¶
- edges become clearer
- the data points at the range 0 to 200 investors are dense
- the cluster at the high rate level is less dense than that of the lower level
- trends begin to appear
- data points have more-than-average number of investors at regular intervals of the rate
- the number goes to 1,200 at mid-level rate
- data points have more-than-average number of investors at regular intervals of the rate
Disscussion on some of the relationships observed in this part of the investigation¶
how did the feature(s) of interest vary with other features in the dataset?¶
BorrowerRate vs Term¶
- an overwhelming majority of loans of all terms are in the range 0.05% to 0.35%
- majority of the tenure of the loans is 3 years or more
- the correlation co-efficient is c. 0.02; it shows very weak, positive correlation
BorrowerRate vs BorrowerAPR¶
- there is a positive correlation between
BorrowerAPRandBorrowerRate - the positive correlation is very strong (c. 0.99)
BorrowerRate vs StatedMonthlyIncome¶
- the upper bound at the 0.35% rate can be seen
- the region of high income and low rates is quite dense; the number of data points here apprears to be more than average
- the correlation co-efficient is c. -0.09; it shows very weak, negative correlation
BorrowerRate vs EmploymentStatusDuration¶
- the upper bound at the 0.35% rate can be seen
- the lower bound is not quite as prominent as the upper one
- the correlation co-efficient is c. 0.02; it shows very weak, positive correlation
PercentFunded vs BorrowerRate¶
PercentFundedis clustered at the range 0.7 (70% funding) to 1 (100% funding) for all levels ofBorrowerRate- the data points are all over the place, so to speak
- there is very weak, negative correlation (c. -0.03)
PercentFunded vs Investors¶
Investorsis clustered at the range 0 to c. 400 for all levels ofBorrowerRate- the cluster is densest at the range 0 to 200
- there is weak, negative correlation (c. -0.27)
are there any interesting relationships between the other features (not the main feature(s) of interest)?¶
BorrowerRate vs Term¶
- there are unusually high and low rates at
Term= 36 months
BorrowerRate vs BorrowerAPR¶
- there is interesting behaviour at the extremes
- low effective rates have quite low nominal rates and vice versa
BorrowerRate vs StatedMonthlyIncome¶
- at lower rate levels (0.0% - 0.05%), there are more observations of low income than high income
- data points form lines at 0.35%, 0.325%, 0.29%, 0.15% and 0.1%
BorrowerRate vs EmploymentStatusDuration¶
- the region of low duration, at all rate levels, is quite dense; the number of data points here apprears to be more than average
- at lower rate levels (0.0% - 0.05%), there are more data points at low duration than high duration
- data points form lines at 0.35%, 0.325%, 0.275% and 0.25%
PercentFunded vs BorrowerRate¶
- there is a clear line at 100% funding
- there is a cluster of points at the high rate level beyond which loans are funded only 100%
PercentFunded vs Investors¶
- there apprears to be a lot of data points at the region of low rates and low number of investors
- data points have more-than-average number of investors at regular intervals of the rate
Multivariate Exploration¶
Will follow the method used in univariate exploration¶
In [20]:
#df to use for viz
corr_df = pd.DataFrame([df.BorrowerRate, df.Term, df.BorrowerAPR, df.StatedMonthlyIncome, df.EmploymentStatusDuration]).transpose()
corr_df = corr_df[pd.notnull(corr_df['EmploymentStatusDuration'])]
corr_df.info()
In [21]:
#sub-plots
fig, ax = create_sub()
#correlation matrix
corr = corr_df.corr()
#plot
with suppress(DeprecationWarning, ImportWarning):
sns.heatmap(corr, mask=np.zeros_like(corr, dtype=np.bool), cmap=sns.diverging_palette(255, 32, l=65, center="dark", as_cmap=True), square=True, ax=ax, annot=True, fmt="0.2f", linewidths=.5);
In [14]:
corr
Out[14]:
Findings¶
- there is weak, positive correlation between
TermandBorrowerRate-> c. 0.02TermandStatedMonthlyIncome-> c. 0.03TermandEmploymentStatusDuration-> c. 0.09StatedMonthlyIncomeandEmploymentStatusDuration-> c. 0.07
- there is weak, negative correlation between
BorrowerRateandStatedMonthlyIncome-> c. -0.09BorrowerRateandEmploymentStatusDuration-> c. -0.02TermandBorrowerAPR-> c. -0.02StatedMonthlyIncomeandBorrowerAPR-> c. -0.08EmploymentStatusDurationandBorrowerAPR-> c. -0.01
- there is strong, positive correlation between
BorrowerAPRandBorrowerRate-> c. 0.99
- none of the variables of interest have a strong, negative correlation
In [8]:
#df to use for viz
corr_df = pd.DataFrame([df.BorrowerRate, df.PercentFunded, df.Investors]).transpose()
corr_df.info()
In [10]:
#sub-plots
fig, ax = create_sub()
#correlation matrix
corr = corr_df.corr()
#plot
with suppress(DeprecationWarning, ImportWarning):
sns.heatmap(corr, mask=np.zeros_like(corr, dtype=np.bool), cmap=sns.diverging_palette(255, 32, l=65, center="dark", as_cmap=True), square=True, ax=ax, annot=True, fmt="0.2f", linewidths=.5);
In [11]:
corr
Out[11]:
Findings¶
- none of the variables of interest have a weak, positive correlation
- there is weak, negative correlation between
BorrowerRateandPercentFunded-> c. -0.03BorrowerRateandInvestors-> c. -0.27PercentFundedandInvestors-> c. -0.05
- none of the variables of interest have a strong, positive correlation
- none of the variables of interest have a strong, negative correlation
Discussion on some of the relationships observed in this part of the investigation¶
are there features that strengthened each other in terms of looking at the feature(s) of interest?¶
- yes and no
- no, the investigation shows that the features of interest do not affect
BorrowerRateas much as expected - yes, the investigation shows that the features of interest, sans
BorrowerRatedo not affect each other significantly
- no, the investigation shows that the features of interest do not affect
- ditto, for features affecting
Investors
are there any interesting or surprising interactions between features?¶
- no; there are no surprises, yet
Summary of Findings¶
Loans¶
- average amount borrowed is c. USD 8,337.00; the range is USD 1,000.00 to 35,000.00
- loans of amount USD 5k, 10k, 15k, 20k and 25k have higher-than-average frequency
- average rate is c. 0.19%; the range is zero to 0.5%
- the number of loans disbursed changes with time
- the number fluctuates
- there are seemingly regular but wide fluctuations
- fluctuations in the periods Q4 2013 to Q2 2014 inclusive are remarkably wide
- there are little to no fluctuations starting Q4 2008 to Q3 2009, inclusive (this is exactly 12 months)
- there are seemingly regular but wide fluctuations
- the number of loans issued appears to increase exponentially despite the fluctuations
- the average number of loans issued, per annum, between 2005 and 2014 inclusive is c. 4.3 million
- the number fluctuates
- an overwhelming majority of loans of all terms are in the range 0.05% to 0.35% borrowing rate
- majority of the tenure of the loans is 3 years or more
- the least funded loans are at 70% funding for all levels of borrowing rate
Occupations and employment status¶
- an overwhelming majority of the top ten occupations are white collar
- a majority of the bottom ten occupations, that are not students, are blue collar; half of the occupations in the bottom ten are students
- Prosper lends to unemployed applicants
Income level¶
- Prosper appears to prefer to lend to high income earners at less-than-average rates
- the region of high income and low rates is quite dense
- an overwhelming majority of loans are issued to those whose income level is above USD 25,000.00 p.a.
- the top 2 groups account for 55.8% of loans issued
- the top 4 groups account for 85.4% of loans issued
Home ownership status¶
- there is an almost-even split between those that own a home and those that do not; 50.4% of borrowers own a home
Location of borrower¶
- a majority of the states with the least borrowers are from the so-called "fly-over country"
- all of the states with the most borrowers are on either seaboard
Factors that affect the nominal borrowing rate¶
- there is weak, positive correlation between
BorrowerRateandTerm-> c. 0.02 - there is weak, negative correlation between
BorrowerRateandStatedMonthlyIncome-> c. -0.09BorrowerRateandEmploymentStatusDuration-> c. -0.02
- there is strong, positive correlation between
BorrowerRateandBorrowerAPR-> c. 0.99
- none of the variables of interest have a strong, negative correlation
Factors that affect the number of investors¶
- none of the variables of interest have a weak, positive correlation
- there is weak, negative correlation between
InvestorsandBorrowerRate-> c. -0.27InvestorsandPercentFunded-> c. -0.05
- none of the variables of interest have a strong, positive correlation
- none of the variables of interest have a strong, negative correlation
Conclusions¶
- Prosper's customer is, mostly, a white collar worker who:
- earns at least USD 25,000.00 a year
- may or may not own a home
- lives on either seaboard
- applies for a c. USD 8,400.00 loan
- to re-pay in at least 3 years
- at c. 0.19% p.a. nominal
- the factors investigated do not affect the borrowing rate significantly; a more extensive and rigorous investigation must be done
- the factors investigated do not affect the proportion of loans invested in significantly; a more extensive and rigorous investigation must be done
In [1]:
from subprocess import call
call(['python', '-m', 'nbconvert', 'Wrangle_and_Analyse_Data.html'])
Out[1]:
In [ ]: